The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Artificial intelligence is set to be deployed in operating rooms to improve surgical care. This early-stage clinical evaluation shows the feasibility of concurrently attaining real-time, high-quality predictions from several deep neural networks for endoscopic video analysis deployed for assistance during three laparoscopic cholecystectomies.

Assessing the critical view of safety in laparoscopic cholecystectomy requires accurate identification and localization of key anatomical structures, reasoning about their geometric relationships to one another, and determining the quality of their exposure. In this work, we propose to capture each of these aspects by modeling the surgical scene with a disentangled latent scene graph representation, which we can then process using a graph neural network. Unlike previous approaches using graph representations, we explicitly encode in our graphs semantic information such as object locations and shapes, class probabilities and visual features. We also incorporate an auxiliary image reconstruction objective to help train the latent graph representations. We demonstrate the value of these components through comprehensive ablation studies and achieve state-of-the-art results for critical view of safety prediction across multiple experimental settings.

One of the recent advances in surgical AI is the recognition of surgical activities as triplets of (instrument, verb, target). Albeit providing detailed information for computer-assisted intervention, current triplet recognition approaches rely only on single frame features. Exploiting the temporal cues from earlier frames would improve the recognition of surgical action triplets from videos. In this paper, we propose Rendezvous in Time (RiT) - a deep learning model that extends the state-of-the-art model, Rendezvous, with temporal modeling. Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition. We validate our proposal on the challenging surgical triplet dataset, CholecT45, demonstrating an improved recognition of the verb and triplet along with other interactions involving the verb such as (instrument, verb). Qualitative results show that the RiT produces smoother predictions for most triplet instances than the state-of-the-arts. We present a novel attention-based approach that leverages the temporal fusion of video frames to model the evolution of surgical actions and exploit their benefits for surgical triplet recognition.

近年来，随着深度神经网络方法的普及，手术计算机视觉领域经历了相当大的突破。但是，用于培训的标准全面监督方法需要大量的带注释的数据，从而实现高昂的成本；特别是在临床领域。已经开始在一般计算机视觉社区中获得吸引力的自我监督学习（SSL）方法代表了对这些注释成本的潜在解决方案，从而使仅从未标记的数据中学习有用的表示形式。尽管如此，SSL方法在更复杂和有影响力的领域（例如医学和手术）中的有效性仍然有限且未开发。在这项工作中，我们通过在手术计算机视觉的背景下研究了四种最先进的SSL方法（Moco V2，Simclr，Dino，SWAV），以解决这一关键需求。我们对这些方法在cholec80数据集上的性能进行了广泛的分析，以在手术环境理解，相位识别和工具存在检测中为两个基本和流行的任务。我们检查了它们的参数化，然后在半监督设置中相对于训练数据数量的行为。如本工作所述和进行的那样，将这些方法的正确转移到手术中，可以使SSL的一般用途获得可观的性能 - 相位识别率高达7％，而在工具存在检测方面，则具有20％ - 半监督相位识别方法高达14％。该代码将在https://github.com/camma-public/selfsupsurg上提供。

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of <instrument, verb, target> combination delivers comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms by competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.

Recent advancements in deep learning methods bring computer-assistance a step closer to fulfilling promises of safer surgical procedures. However, the generalizability of such methods is often dependent on training on diverse datasets from multiple medical institutions, which is a restrictive requirement considering the sensitive nature of medical data. Recently proposed collaborative learning methods such as Federated Learning (FL) allow for training on remote datasets without the need to explicitly share data. Even so, data annotation still represents a bottleneck, particularly in medicine and surgery where clinical expertise is often required. With these constraints in mind, we propose FedCy, a federated semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos, thereby improving performance on the task of surgical phase recognition. By leveraging temporal patterns in the labeled data, FedCy helps guide unsupervised training on unlabeled data towards learning task-specific features for phase recognition. We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases using a newly collected multi-institutional dataset of laparoscopic cholecystectomy videos. Furthermore, we demonstrate that our approach also learns more generalizable features when tested on data from an unseen domain.

为有效语义分割和特别是视频语义分割构建模型的主要障碍是缺乏大型和良好的注释数据集。这种瓶颈在高度专业化的和监管领域特别禁止，例如医学和手术，视频语义细分可能具有重要应用，但数据和专家注释是稀缺的。在这些设置中，可以在培训期间利用时间线索和解剖结构来提高性能。在这里，我们呈现时间限制的神经网络（TCNN），是用于外科视频的视频语义分割的半监督框架。在这项工作中，我们表明AutoEncoder网络可用于有效地提供空间和时间监控信号来培训深度学习模型。我们在新推出的腹腔镜胆囊切除术文程序，内测序和对CADIS，CADIS的公共数据集的适应时测试我们的方法。我们证明，可以利用预测面罩的较低尺寸表示，以在稀疏标记的数据集中提供一致的改进，这些数据集在推理时间不具有额外的计算成本。此外，TCNN框架是模型无关的，可以与其他模型设计选择结合使用，具有最小的额外复杂性。

医疗AI通过支持基于证据的医学实践，个性化患者治疗，降低成本以及改善提供者和患者体验，推进医疗保健的巨大潜力。我们认为解锁此潜力需要一种系统的方法来衡量在大规模异构数据上的医疗AI模型的性能。为了满足这种需求，我们正在建立Medperf，这是一个开放的框架，用于在医疗领域的基准测试机器学习。 Medperf将使联合评估能够将模型安全地分配给不同的评估设施，从而赋予医疗组织在高效和人类监督过程中评估和验证AI模型的性能，同时优先考虑隐私。我们描述了当前的挑战医疗保健和AI社区面临，需要开放平台，Medperf的设计理念，其目前的实施状态和我们的路线图。我们呼吁研究人员和组织加入我们创建Medperf开放基准平台。

临床医生在手术室（OR）的细粒度定位是设计新一代或支持系统的关键组成部分。需要基于人像素的分段和身体视觉计算机的计算机视觉模型检测，以更好地了解OR的临床活动和空间布局。这是具有挑战性的，这不仅是因为或图像与传统视觉数据集有很大不同，还因为在隐私问题上很难收集和生成数据和注释。为了解决这些问题，我们首先研究了如何在低分辨率图像上进行姿势估计和实例分割，而下采样因子从1x到12倍进行下采样因子。其次，为了解决域的偏移和缺乏注释，我们提出了一种新型的无监督域适应方法，称为适配器，以使模型从野外标记的源域中适应统计上不同的未标记目标域。我们建议在未标记的目标域图像的不同增强上利用明确的几何约束，以生成准确的伪标签，并使用这些伪标签在自我训练框架中对高分辨率和低分辨率或图像进行训练。此外，我们提出了分离的特征归一化，以处理统计上不同的源和目标域数据。对两个或数据集MVOR+和TUM-或TUM-或测试的详细消融研究的广泛实验结果表明，我们方法对强构建的基线的有效性，尤其是在低分辨率的隐私性或图像上。最后，我们在大规模可可数据集上显示了我们作为半监督学习方法（SSL）方法的普遍性，在这里，我们获得了可比较的结果，而对经过100％标记的监督培训的模型的标签监督只有1％。 。